Method for finding paths in video

ABSTRACT

A system for detecting behavior of a target may include: a target detection engine, adapted to detect at least one target from one or more objects from a video surveillance system recording a scene; a path builder, adapted to create at least one mature path model from analysis of the behavior of a plurality of targets in the scene, wherein the at least one mature path model includes a model of expected target behavior with respect to the at least one path model; and a target behavior analyzer, adapted to analyze and identify target behavior with respect to the at least one mature path model. The system may further include an alert generator, adapted to generate an alert based on the identified behavior.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation in part of U.S. application Ser. No.10/948,751, entitled “METHOD FOR FINDING PATHS IN VIDEO,” filed Sep. 24,2004, the contents of which are incorporated herein in their entirety.

FIELD OF THE INVENTION

The present invention is related to video-based surveillance andmonitoring. More specifically, specific embodiments of the inventionrelate to context-sensitive video-based surveillance and monitoringsystems, with applications in market research and/orstatistical/contextual target modeling.

BACKGROUND OF THE INVENTION

Many businesses and other facilities, such as banks, stores, airports,etc., make use of security systems. Among such systems are video-basedsystems, in which a sensing device, like a video camera, obtains andrecords images within its sensory field.

For example, a video camera will provide a video record of whatever iswithin the field-of-view of its lens. Such video images may be monitoredby a human operator and/or reviewed later by a human operator. Recentprogress has allowed such video images to be monitored also by anautomated system, improving detection rates and saving human labor.

In many situations it would be desirable to specify the detection oftargets using relative modifiers such as fast, slow, tall, flat, wide,narrow, etc., without quantifying these adjectives. Likewise it would bedesirable for state-of-the-art surveillance systems to adapt to thepeculiarities of the scene, as current systems are unable to do so, evenif the same systems have been monitoring the same scene for many years.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed to enabling theautomatic extraction and use of contextual information. Furthermore,embodiments of the present invention may provide contextual informationabout moving targets. This contextual information may be used to enablecontext-sensitive event detection, and it may improve target detection,improve tracking and classification, and decrease the false alarm rateof video surveillance systems.

The embodiments of the invention may include a system that builds pathmodels from analysis of a plurality of targets observed from asurveillance video sequence. The mature path models may be used toidentify whether a target's behavior is consistent with respect to theexpected target behavior, to predict a target's subsequent path based onthe target's observed behavior and to classify a target's type. Theembodiments of the invention may also include building a statisticalmodel of targets' behavior with respect to their path models, which maybe used to analyze a target's interaction with scene elements and withother targets.

A method of video processing may include automatic extraction and use ofcontextual information about moving targets in a surveillance video. Thecontextual information may be gathered in the form of statistical modelsrepresenting the expected behavior of targets. These models may be usedto detect context sensitive events when a target's behavior does notconform to the expected behavior. Furthermore, detection, tracking andclassification of targets may also be improved using the contextualinformation.

In one embodiment, a system for detecting behavior of a target, mayinclude: a target detection engine, adapted to detect at least onetarget from one or more objects from a video surveillance systemrecording a scene; a path builder, adapted to create at least one maturepath model from analysis of the behavior of a plurality of targets inthe scene, wherein the at least one mature path model includes a modelof expected target behavior with respect to the at least one path model;a target behavior analyzer, adapted to analyze and identify targetbehavior with respect to the at least one mature path model; and analert generator, adapted to generate an alert based on the identifiedbehavior.

In another embodiment, a computer-based method of target behavioranalysis may include the steps of: processing an input video sequence toobtain target information for at least one target from one or moreobjects from a video surveillance system recording a scene; building atleast one mature path model from analysis of the behavior of a pluralityof targets in the scene, wherein the at least one mature path modelincludes a model of expected target behavior with respect to the atleast one path model; analyzing and identifying target behavior of atarget with respect to the at least one mature path model; andgenerating an alert based on the identified target behavior.

In another embodiment, a computer-readable medium may containinstructions that, when executed by a processor, cause the processor toperform operations including:

processing an input video sequence to obtain target information for atleast one target from one or more objects from a video of a scene;building at least one mature path model from analysis of the behavior ofa plurality of targets in the scene, wherein said at least one maturepath model includes a model of expected target behavior with respect tosaid at least one path model; and analyzing and identifying targetbehavior of a target with respect to said at least one mature pathmodel.

The invention may be embodied in the form of hardware, software, orfirmware, or in the form of combinations thereof.

DEFINITIONS

The following definitions are applicable throughout this disclosure,including in the above.

A “video” may refer to motion pictures represented in analog and/ordigital form.

Examples of video include: television, movies, image sequences from avideo camera or other observer, and computer-generated image sequences.

A “frame” may refer to a particular image or other discrete unit withina video.

An “object” may refer to an item of interest in a video. Examples of anobject include: a person, a vehicle, an animal, and a physical subject.

A “target” may refer to a computer's model of an object. A target may bederived via image processing, and there is a one-to-one correspondencebetween targets and objects.

A “target instance,” or “instance,” may refer to a sighting of an objectin a frame.

An “activity” may refer to one or more actions and/or one or morecomposites of actions of one or more objects. Examples of an activityinclude: entering; exiting;

stopping; moving; raising; lowering; growing; and shrinking.

A “location” may refer to a space where an activity may occur. Alocation may be, for example, scene-based or image-based. Examples of ascene-based location include: a public space; a store; a retail space;an office; a warehouse; a hotel room; a hotel lobby; a lobby of abuilding; a casino; a bus station; a train station; an airport; a port;a bus; a train; an airplane; and a ship. Examples of an image-basedlocation include: a video image; a line in a video image; an area in avideo image; a rectangular section of a video image; and a polygonalsection of a video image.

An “event” may refer to one or more objects engaged in an activity. Theevent may be referenced with respect to a location and/or a time.

A “computer” may refer to one or more apparatus and/or one or moresystems that are capable of accepting a structured input, processing thestructured input according to prescribed rules, and producing results ofthe processing as output. Examples of a computer may include: acomputer; a stationary and/or portable computer; a computer having asingle processor, multiple processors, or multi-core processors, whichmay operate in parallel and/or not in parallel; a general purposecomputer; a supercomputer; a mainframe; a super mini-computer; amini-computer; a workstation; a micro-computer; a server; a client; aninteractive television; a web appliance; a telecommunications devicewith internet access; a hybrid combination of a computer and aninteractive television; a portable computer; a personal digitalassistant (PDA); a portable telephone; application-specific hardware toemulate a computer and/or software, such as, for example, a digitalsignal processor (DSP), a field-programmable gate array (FPGA), a chip,chips, or a chip set; an optical computer; a quantum computer; abiological computer; and an apparatus that may accept data, may processdata in accordance with one or more stored software programs, maygenerate results, and typically may include input, output, storage,arithmetic, logic, and control units.

A “computer-readable medium” may refer to any storage device used forstoring data accessible by a computer. Examples of a computer-readablemedium may include: a magnetic hard disk; a floppy disk; an opticaldisk, such as a CD-ROM and a DVD; a magnetic tape; and a memory chip.

“Software” may refer to prescribed rules to operate a computer. Examplesof software may include: software; code segments; instructions; applets;pre-compiled code; compiled code; computer programs; and programmedlogic.

A “computer system” may refer to a system having one or more computers,where each computer may include a computer-readable medium embodyingsoftware to operate the computer. Examples of a computer system mayinclude: a distributed computer system for processing information viacomputer systems linked by a network; two or more computer systemsconnected together via a network for transmitting and/or receivinginformation between the computer systems; and one or more apparatusesand/or one or more systems that may accept data, may process data inaccordance with one or more stored software programs, may generateresults, and typically may include input, output, storage, arithmetic,logic, and control units.

A “network” may refer to a number of computers and associated devicesthat may be connected by communication facilities. A network may involvepermanent connections such as cables or temporary connections such asthose made through telephone or other communication links. Examples of anetwork may include: an internet, such as the Internet; an intranet; alocal area network (LAN); a wide area network (WAN); and a combinationof networks, such as an internet and an intranet.

A “sensing device” may refer to any apparatus for obtaining visualinformation. Examples include: color and monochrome cameras, videocameras, closed-circuit television (CCTV) cameras, charge-coupled device(CCD) sensors, analog and digital cameras, PC cameras, web cameras, andinfra-red imaging devices. If not more specifically described, a“camera” refers to any sensing device.

A “blob” may refer generally to any object in an image (usually, in thecontext of video). Examples of blobs include moving objects (e.g.,people and vehicles) and stationary objects (e.g., bags, furniture andconsumer goods on shelves in a store).

A “target property map” may refer to a mapping of target properties orfunctions of target properties to image locations. Target property mapsare built by recording and modeling a target property or function of oneor more target properties at each image location. For instance, a widthmodel at image location (x,y) may be obtained by recording the widths ofall targets that pass through the pixel at location (x,y). A model maybe used to represent this record and to provide statistical information,which may include the average width of targets at location (x,y), thestandard deviation from the average at this location, etc. Collectionsof such models, one for each image location, are called a targetproperty map.

A “path” may refer to an image region, not necessarily connected, thatrepresents the loci of targets: a) whose trajectories start near thestart point of the path; b) whose trajectories end near the end point ofthe path; and c) whose trajectories overlap significantly with the path.

BRIEF DESCRIPTION OF THE DRAWINGS

Specific embodiments of the invention will now be described in furtherdetail in conjunction with the attached drawings, in which:

FIG. 1 depicts a flowchart of a content analysis system that may includeembodiments of the invention;

FIG. 2 depicts a flowchart describing training of paths, according to anembodiment of the invention;

FIG. 3 depicts a flowchart describing the training of target propertymaps according to an embodiment of the invention;

FIG. 4 depicts a flowchart describing the use of target property mapsaccording to an embodiment of the invention;

FIG. 5 depicts a block diagram of a system that may be used inimplementing some embodiments of the invention; and

FIG. 6 depicts a block diagram of a system according to embodiments ofthe present invention.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS OF THE INVENTION

Embodiments of the invention may comprise part of a general surveillancesystem. A potential embodiment is illustrated in FIG. 1. Target propertyinformation is extracted from the video sequence by detection (11),tracking (12) and classification (13) modules. These modules may utilizeknown or as yet to be developed techniques. The target propertyinformation may be extracted from live video or from previously recordedvideo. The resulting information is passed to an event detection module(14) that matches observed target properties against properties deemedthreatening by a user. For example, the user may be able to specify suchthreatening properties by using a graphical user interface (GUI) (15) orother input/output (I/O) interface with the system. The path builder(16) monitors and models the data extracted by the up-stream components(11), (12), and (13), and it may further provide information to thosecomponents. Data models may be based on target properties, which mayinclude, but which are not limited to, the target's location, width,height, size, speed, direction-of-motion, time of sighting, age, etc.This information may be further filtered, interpolated and/orextrapolated to achieve spatially and temporally smooth and continuousrepresentations.

LEARNING PATHS BY OBSERVATION

According to some embodiments of the invention, paths may need to belearned by observation before the paths can be used. To signal thevalidity of a path model, the path model is labeled “mature” only aftera statistically meaningful amount of data has been observed. Queries topath models that have not yet matured are not answered. This strategyleaves the system in a default mode until at least some of the modelshave matured. When a path model has matured, it may provide informationthat may be incorporated into the decision making processes of connectedalgorithmic components.

The availability of this additional information may help the algorithmiccomponents to make better decisions.

Not all targets or their instances are necessarily used for training.The upstream components (11), (12), and (13) that gather targetproperties may fail, and it is important that the models are shieldedfrom data that is faulty. One technique for dealing with this problem isto devise algorithms that carefully analyze the quality of the targetproperties.

In other embodiments of the invention, a simple algorithm may be usedthat rejects targets and target instances if there is a doubt abouttheir quality. This latter approach likely extends the time until targetproperty maps achieve maturity. However, the prolonged time that manyvideo surveillance systems spend viewing a scene makes this optionattractive in that the length of time to maturity is not likely to beproblematic.

An overview of an exemplary method for learning path models according toan embodiment of the invention is shown in FIG. 2. The major componentsmay include initialization of the path model (201), training of sizemaps (202), training of entry/exit maps (203), and training of pathmodels (204).

Size maps may be generated in Block 202 and may be used by theentry/exit map training algorithm (203) to associate trajectories withentry/exit regions. Entry/exit regions that are close compared to thenormal size of the targets that pass through them are merged. Otherwisethey are treated as separate entry/exit regions.

Entry/exit maps, which may be generated in Block 203, may in turn formthe basis for path models. When entry/exit regions have matured they canbe used to measure target movement statistics between them. Thesestatistics may be used to form the basis for path models in Block 204.

The size and entry/exit maps are types of target property maps, and theymay be trained (built) using a target property map training algorithm,which is described in co-pending, commonly-assigned U.S. Publication No.2006-0072010A1 (U.S. patent application Ser. No. 10/948,785), filed onSep. 24, 2004, entitled, “Target Property Maps for SurveillanceSystems,” and incorporated herein by reference. The target property maptraining algorithm may be used several times in the process shown inFIG. 2. To simplify the description of this process, the target propertymap training algorithm is explained here in detail and then referencedlater in the algorithm detailing the extraction of path models.

FIG. 3 depicts a flowchart of an algorithm for building target propertymaps, according to an embodiment of the invention. The algorithm maybegin by appropriately initializing an array corresponding to the sizeof the target property map (in general, this may correspond to an imagesize) in Block 301. In Block 302, a next target may be considered. Thisportion of the process may begin with initialization of a buffer, whichmay be a ring buffer, of filtered target instances, in Block 303. Theprocedure may then proceed to Block 304, where a next instance (whichmay be stored in the buffer) of the target under consideration may beaddressed. In Block 305, it is determined whether the target isfinished; this is the case if all of the target's instances have beenconsidered. If the target is finished, the process may proceed to Block309 (to be discussed below). Otherwise, the process may then proceed toBlock 306, to determine if the target is bad; this is the case if thislatest instance reveals a severe failure of the target's handling,labeling or identification by the up-stream processes. If this is thecase, the process may loop back to Block 302, to consider the nexttarget. Otherwise, the process may proceed with Block 307, to determineif the particular instance under consideration is a bad instance; thisis the case if the latest instance reveals a limited inconsistency inthe target's handling, labeling or identification by the up-streamprocess. If a bad instance was found, that instance is ignored and theprocess proceeds to Block 304, to consider the next target instance.Otherwise, the process may proceed with Block 308 and may update thebuffer of filtered target instances, before returning to Block 304, toconsider the next target instance.

Following Block 305 (as discussed above), the algorithm may proceed withBlock 309, where it is determined which, if any, target instances may beconsidered to be “mature.” According to an embodiment of the invention,if the buffer is found to be full, the oldest target instance in thebuffer may be marked “mature.” If all instances of the target have beenconsidered (i.e., if the target is finished), then all target instancesin the buffer may be marked “mature.”

The process may then proceed to Block 310, where target property mapmodels may be updated at the map locations corresponding to the maturetarget instances.

Following this map updating, the process may determine, in Block 311,whether or not each model is mature. In particular, if the number oftarget instances for a given location is larger than a preset number ofinstances required for maturity, the map location may be marked“mature.” As discussed above, only mature locations may be used inaddressing inquiries.

Returning, now, to the process of FIG. 2, the target property maptraining algorithm of FIG. 3 will be referenced in describing theprocess of training path models. As discussed above, in Block 201, apath model may be initialized at the outset of the process. This may bedone, for example, by initializing an array, which may be the size of animage (e.g., of a video frame).

The process of FIG. 2 may then proceed to Block 202, training of sizemaps. In an embodiment of the invention, the process of Block 202 usesthe target property map training algorithm of FIG. 3 to train one ormore size maps. The generic target property training algorithm of FIG. 3may be changed to perform this particular type of training by modifyingBlocks 301, 308, and 310. All three of these blocks, in Block 202 ofFIG. 2, operate on size map instances of the generic target property mapobjects. Component 308 extracts size information from the targetinstance stream that enters the path builder (component 16 in FIG. 1).Separate size maps may be maintained for each target type and forseveral time ranges.

The process of FIG. 2 may then train entry/exit region maps (Block 203).Once again, the algorithm of FIG. 3 may be used to perform the maptraining. To do so, the instantiations of the initialization component(301), the extraction of target origin and destination information(308), and the target property model update component (310) may all bechanged to suit this particular type of map training. Component 301 mayoperate on entry/exit map instances of the generic target property mapobjects. Component 308 may extract target scene entry and exitinformation from the target instance stream that enters the path builder(component 16 in FIG. 1). Component 309 may determine a set of entry andexit regions that represent a statistically significant number oftrajectories. These regions are deemed to deserve representation and maybe annotated with target statistics, such as, but not limited to, theregion size and location, the percentage of targets in the scene thatenter or exit through the region, etc. Component 310 may update theentry/exit region model to reflect changes to the shapes and/or targetcoverage of the entry/exit regions. This process may use informationprovided by a size map trained in Block 202 to decide whether adjacententry or exit regions need to be merged. Entry regions that are close toeach other may be merged into a single region if the targets that usethem are large compared to the distance between them. Otherwise, theymay remain separate regions. The same approach may be used for exitregions. This enables maintaining separate paths even when the targetson them appear to be close to each other at a great distance from thecamera. The projective transformation that controls image formation isthe cause for the apparent close proximity of distant objects. One mayuse the ratio of target size over entry/exit region distance,

$\frac{{target}\mspace{14mu} {size}}{{distance}\mspace{14mu} {between}\mspace{14mu} {regions}},$

target size/distance between regions, for example, as it is practicallyinvariant under perspective transformation and thus simplifies theregion maintenance algorithm. Separate size maps may be maintained foreach target type and for several time ranges.

Path models may then be trained, Block 204. According to an embodimentof the invention, this may begin with initialization of a path datastructure. The process may then use the information contained in theentry and exit region map to build a table with a row for each entryregion and a column for every exit region in the entry and exit regionmap. Each trajectory may be associated with an entry region from whichit originates and an exit region where it terminates. The set oftrajectories associated with an entry/exit region pair is used to definethe locus of the path. According to various embodiments of theinvention, a path may be determined by taking the intersection of alltrajectories in the set, by taking the union of those trajectories, orby defining a path to correspond to some minimum percentage oftrajectories in the set. The path data structure combines theinformation gathered about each path: the start and end points of thepath, the number or fraction of trajectories it represents, and twoindices into the entry/exit region map that indicate which entry andexit regions in that data structure it corresponds to. Separate pathmodels may be maintained for each type of target and for several timeranges.

USING PATH MODELS

Path models may be obtained and maintained using information from anexisting surveillance system. However, to make path models useful, thepath models must also be able to provide information to the system. Pathmodels may allow prediction of a target's destination, given thetarget's location and its observed trajectory. For example, a targetpath in a path model for a hardware store may describe that targetsleaving the power-tools department tend to stop at the departmentcheck-out. In another example, a target path in a path model maydescribe that targets traveling the path tend to reach the other end ofthe path within a specific time frame, e.g., two minutes.

Path models may also allow classification of a target's path or of thetarget, based on the path type. For example, targets that are vehicles,pedestrians, trains or airplanes tend to travel, respectively, on roads,sidewalks, railroad tracks or runways.

FIG. 6 depicts a block diagram of a system for creating and using pathmodels, according to embodiments of the present invention. The systemmay include a target detection engine 602. Target detection engine 602may detect one or more targets 604 from one or more objects from a videosurveillance system recording a scene (not shown). Targets 604 may beprovided to path builder 16 for the creation of a mature path model 606,as described above. Mature path model 606 may include a model ofexpected target behavior with respect to the mature path model. A targetbehavior analyzer 608 may analyze and identify behavior of laterdetected targets 604 with respect to the mature path model 606. An alertgenerator 610 may receive the results of the analysis and may generatean alert 612 when a specified behavior is detected and identified.Examples of the use of the system are illustrated below.

USING PATH MODELS IN MARKET RESEARCH AND/OR STATISTICAL/CONTEXTUALTARGET MODELING

Path models may also allow analysis of target properties. In anexemplary embodiment, market research and/or statistical/contextualtarget modeling may benefit from the following information determinedfrom path models.

Information about target dwell times and locations along learned pathsmay help to determine, e.g., where shoppers spend their time whileon-site, on which aisle and/or in front of which products, whichproducts customers compare, and which products they select with orwithout comparison to other products.

Information about relative dwell locations along learned paths may helpto determine, e.g., whether customers that were interested in product Aalso look at product B and with what probability C and dwell time D.

Information about target properties associated with paths, dwelllocations and times may help to associate, for example, a target typewith a target size, or a target's clothing or uniform.

Information about interactions of targets on paths with other targets ofthe same or different type may help detection, for example, of whenvehicles stop next to each other while traveling to and from a sensitivesite.

Information about interactions of targets on a path with scene elements,such as, e.g., buildings, roads, sidewalks, grass/lawn regions, and/orwater regions, may help to determine, for example, how many (distinct)customers make use of an aisle-barcode reader, or how many vehiclesactually stop at a four-way-stop intersection.

Information about temporal patterns of target properties on a path, suchas weekday vs. weekend, morning vs. noon vs. evening vs. nighttime,summer vs. winter, may help with determining normal building accesspatterns after-hours for security applications.

Information about deviations from normal target properties along a pathdue to time of day/week/year, location, target type, and/or trafficdensity, for instance, normal access pattern information, may help todetermine suspicious building access.

In addition, the information described above may be combined in manyways to provide further benefit to market research and/orstatistical/contextual target modeling.

STATISTICAL MODELING FOR PUBLIC SAFETY AND PLANNING

Gathering statistical data of target behavior on a path may provide arange of target properties on the path, for example, normal speed, size,width, and/or height of moving objects. In one application, lawenforcement may use this information to determine the normal speed,size, width, and/or height of objects moving on e.g., footpaths, parkinglots, roads, water channels, canals, lakes, ports, and/or airporttaxiways/runways. The statistical information can be used further todetermine deviations from normal object properties in subsequentlyobserved targets.

Gathering statistical data of target behavior on a path may provide arange of, for example, normal driving regions, directions, object entryand exit probabilities. In one application, for example, trafficplanning, reconnaissance or surveillance applications may use thisinformation to determine traffic statistics that can highlight, e.g.,choke points, popular access points, underutilized access points, and/ortraffic patterns.

Gathering statistical data of target behavior on a path may providehigher order statistics of objects. For instance, traffic planners mayuse this information to determine the expected deviation from normalobject behavior. This information can be used further to determinedeviations from the expected deviation from normal object behavior.

ANALYSIS AND DETECTION OF UNUSUAL TARGET BEHAVIOR ON A PATH

Path models may also allow detection of unusual target properties and/orbehavior, such as, for example, when a target deviates from its path.For instance, information about a target's deviation from a path mayhelp to detect targets that travel in parts of the scene not associatedwith any known path, or to detect targets that enter the scene outsideknown entry points/regions and/or known exit points/regions. In anotherexample, a target leaving a path at a point other than the exitpoint/region expected for targets on the path may be detected. Thisinformation may help to detect, for example, vehicles that fail totravel between designated checkpoints.

Deviation from a path may also be determined by detection of a failureto arrive on time or at the desired location. For instance, security andsurveillance applications may use this information to determine whethera person or vehicle passes swiftly and directly between checkpoints. Inproduction process monitoring, this information may be used to determinewhether a manufacturing process is functioning as intended.

In another example, a target joining a path at a point other than theentry point/region expected for targets on the path may be detected.This information may help to detect, for example, customers leaving thepremises of a shop without passing a checkout or service desk.

Information about a target switching paths may help to detect, forexample, targets that travel first on an employee or customer or visitorpath, and then switch to a path associated with security guards.

Information about a target crossing a path may help to detect, forexample, vehicles in a parking lot (each starting from mutually disjointworld locations), that are expected to merge into the exit lanes, ratherthan crossing them.

Information about a target traveling on an infrequently used path mayhelp to detect, for example, access to a dangerous area at a refinery.

Information about a target traveling unusually slowly, unusually fast orstopping where targets do not usually stop may help to detect, forexample, vehicles that stop between border checkpoints of neighboringcountries. In traffic monitoring applications, this information may helpto detect vehicles traveling above the speed limit.

Information about a target traveling on a path, but at an unusual time,may help to detect, for example, unauthorized access to a closedfacility at nighttime, even if the same facility is accessible by day.This information may also allow the comparison of current targetbehavior with access patterns normal for a particular time of day todetect potential trespassers.

Information about a target traveling on a path, but in unusualdirection, may help to detect, for example, “ghost drivers” traveling inthe wrong direction along a highway.

In another example, this information may be used to determine that theheading of a target heading is going to bring the target too close to asensitive site.

Information about a target traveling on a path that is not normallyassociated with targets of the target's type may help to detect, forexample, vehicles on a sidewalk or an urban pedestrian area.

Information about properties of the target on a certain path that areunusual may help to detect targets whose width, height, size, area,target perimeter length, color (hue, saturation, luminance), texture,compactness, shape and/or time of appearance is unexpected.

In addition, the information described above may be combined in manyways to provide further benefit to detection of dangerous, unauthorized,suspicious, or otherwise noteworthy behavior.

Information about two or more events may be combined to detect unusualco-occurrences. One or more detected unusual target behaviors may becombined, or with target behaviors detected in the context of astatistical model to detect unusual co-occurrences. For instance,surveillance applications may use information of a detected site accesswith detection of an un-manned guard post to detect an unauthorizedaccess.

FIG. 4 depicts a flowchart of an algorithm for querying path models(e.g., by one or more components of a surveillance system) to obtaincontextual information, according to an embodiment of the invention.

The algorithm of FIG. 4 may begin by considering a next target, in Block41. It may then proceed to Block 42, to determine if the requested pathmodel has been defined.

If not, the information about the target is unavailable, and the processmay loop back to Block 41, to consider a next target.

If the requested path model is determined to be available, the processmay then consider a next target instance, in Block 43. If the instanceindicates that the target is finished, in Block 44, the process may loopback to Block 41 to consider a next target. A target is consideredfinished if all of its instances have been considered. If the target isnot finished, the process may proceed to Block 45 and may determine ifthe target property map model at the location of the target instanceunder consideration has matured. If it has not matured, the process mayloop back to Block 43 to consider a next target instance. Otherwise, theprocess may proceed to Block 46, where target context may be updated.The context of a target may be updated by recording the degree of itsconformance with the target property map maintained by this algorithm.Following Block 46, the process may proceed to Block 47 to determinenormalcy properties of the target based on its target context. Thecontext of each target is maintained to determine whether it acted in amanner that is inconsistent with the behavior or observations predictedby the target property map model. Finally, following Block 47, theprocedure may return to Block 41 to consider a next target.

Some embodiments of the invention, as discussed above, may be embodiedin the form of software instructions on a machine-readable medium. Suchan embodiment is illustrated in FIG. 5. The computer system of FIG. 5may include at least one processor 52, with associated system memory 51,which may store, for example, operating system software and the like.The system may further include additional memory 53, which may, forexample, include software instructions to perform various applications.The system may also include one or more input/output (I/O) devices 54,for example (but not limited to), keyboard, mouse, trackball, printer,display, network connection, etc. The present invention may be embodiedas software instructions that may be stored in system memory 51 or inadditional memory 53. Such software instructions may also be stored inremovable or remote media (for example, but not limited to, compactdisks, floppy disks, etc.), which may be read through an I/O device 54(for example, but not limited to, a floppy disk drive). Furthermore, thesoftware instructions may also be transmitted to the computer system viaan I/O device 54 for example, a network connection; in such a case, asignal containing the software instructions may be considered to be amachine-readable medium.

The invention has been described in detail with respect to variousembodiments, and it will now be apparent from the foregoing to thoseskilled in the art that changes and modifications may be made withoutdeparting from the invention in its broader aspects. The invention,therefore, as defined in the appended claims, is intended to cover allsuch changes and modifications as fall within the true spirit of theinvention.

1. A system for detecting behavior of a target, comprising: a targetdetection engine, adapted to detect at least one target from one or moreobjects from a video of a scene; a path builder, adapted to create atleast one mature path model from analysis of the behavior of a pluralityof targets in the scene, wherein said at least one mature path modelincludes a model of expected target behavior with respect to said atleast one path model; and a target behavior analyzer, adapted to analyzeand identify target behavior with respect to said at least one maturepath model.
 2. The system according to claim 1, wherein said targetbehavior analyzer is adapted to identify behavior inconsistent with themodel of expected behavior, comprising detecting at least one of: atarget deviating from the path; a target traveling off of the path; atarget switching from the path to another path; a target crossing thepath; a target traveling the path, wherein the path is infrequentlytraveled; a target traveling at an unusual speed on the path; a targetstopping at an unusual stopping point on the path; a target traveling atan unusual time on the path; a target traveling in an unusual directionon the path; a target of a type not normally found on the path; or atarget having an unusual physical property on the path.
 3. The systemaccording to claim 1, wherein said target behavior analyzer is adaptedto predict a target's subsequent path based on said at least one pathmodel and the target's observed behavior.
 4. The system according toclaim 1, wherein said target behavior analyzer is adapted to classify atleast one of the target type, based on a path type, or the path typebased on the target type.
 5. The system according to claim 1, whereinthe target behavior analyzer is further adapted to build a statisticalmodel of target behavior with respect to the path model.
 6. The systemaccording to claim 5, wherein the target behavior is analyzed withrespect to a statistical model of at least one of: a target dwell timeduration on the path; a target dwell location on the path; a targetproperty associated with at least one of: the path, a dwell location onthe path, and/or a dwell time on the path; an interaction of a targetwith another target of the same type on the path; an interaction of atarget with another target of a different type on the path; aninteraction of a target with an element in the scene; a temporal patternof a target property on the path; or a deviation from normal targetproperties on the path.
 7. The system according to claim 6, wherein thetarget behavior analyzer is adapted to identify target behavior from acombination of: at least two detected behaviors inconsistent with a pathmodel; at least two target detected behaviors with respect to astatistical model; or at least one detected inconsistent behavior and atleast one detected behavior with respect to a statistical model.
 8. Thesystem according to claim 1, wherein said video is received from a videosurveillance system.
 9. The system according to claim 1, wherein saidsystem is implemented in application-specific hardware to emulate atleast one of a computer or software.
 10. The system according to claim1, further comprising an alert generator, adapted to generate an alertbased on said identified behavior.
 11. A computer-based method of targetbehavior analysis, comprising: processing an input video sequence toobtain target information for at least one target from one or moreobjects from a video of a scene; building at least one mature path modelfrom analysis of the behavior of a plurality of targets in the scene,wherein said at least one mature path model includes a model of expectedtarget behavior with respect to said at least one path model; andanalyzing and identifying target behavior of a target with respect tosaid at least one mature path model.
 12. The method according to claim11, further comprising: generating an alert based on said identifiedtarget behavior.
 13. A computer-readable medium containing instructionsthat, when executed by a processor, cause the processor to perform themethod according to claim
 11. 14. A video processing system comprising:a computer system; and the computer-readable medium according to claim13.
 15. A video surveillance system comprising: at least one camera togenerate an input video sequence; and the video processing systemaccording to claim
 14. 16. The method according to claim 11, whereinprocessing an input video sequence comprises processing video from avideo surveillance system.
 17. The method according to claim 11, whereinsaid analyzing and identifying target behavior includes identifyingtarget behavior inconsistent with the model of expected behavior,comprising detecting at least one of: a target deviating from the path;a target traveling off of the path; a target switching from the path toanother path; a target crossing the path; a target traveling the path,wherein the path is infrequently traveled; a target traveling at anunusual speed on the path; a target stopping at an unusual stoppingpoint on the path; a target traveling at an unusual time on the path; atarget traveling in an unusual direction on the path; a target of a typenot normally found on the path; or a target having an unusual physicalproperty on the path.
 18. The method according to claim 11, furthercomprising predicting a target's subsequent path based on said at leastone path model and the target's observed behavior.
 19. The methodaccording to claim 11, wherein said analyzing and identifying targetbehavior includes classifying at least one of the target type, based ona path type, and/or the path type based on the target type.
 20. Themethod according to claim 11, wherein said analyzing and identifyingtarget behavior further includes building a statistical model of targetbehavior with respect to the path model.
 21. The method according toclaim 20, wherein said analyzing and identifying target behaviorincludes analyzing target behavior with respect to a statistical modelof at least one of: a target dwell time duration on the path; a targetdwell location on the path; a target property associated with at leastone of: the path, a dwell location on the path, and/or a dwell time onthe path; an interaction of a target with another target of the sametype on the path; an interaction of a target with another target of adifferent type on the path; an interaction of a target with an elementin the scene; a temporal pattern of a target property on the path; or adeviation from normal target properties on the path.
 22. The methodaccording to claim 21, wherein said analyzing and identifying targetbehavior includes identify target behavior from a combination of: atleast two detected behaviors inconsistent with a path model; at leasttwo target detected behaviors with respect to a statistical model; or atleast one detected inconsistent behavior and at least one detectedbehavior with respect to a statistical model.
 23. A computer-readablemedium containing instructions that, when executed by a processor, causethe processor to perform operations comprising: processing an inputvideo sequence to obtain target information for at least one target fromone or more objects from a video of a scene; building at least onemature path model from analysis of the behavior of a plurality oftargets in the scene, wherein said at least one mature path modelincludes a model of expected target behavior with respect to said atleast one path model; and analyzing and identifying target behavior of atarget with respect to said at least one mature path model.
 24. Thecomputer-readable medium according to claim 23, the operations furthercomprising: generating an alert based on said identified targetbehavior.
 25. The computer-readable medium according to claim 23,wherein processing an input video sequence comprises processing videofrom a video surveillance system.
 26. The computer-readable mediumaccording to claim 23, wherein said analyzing and identifying targetbehavior includes identifying target behavior inconsistent with themodel of expected behavior, comprising detecting at least one of: atarget deviating from the path; a target traveling off of the path; atarget switching from the path to another path; a target crossing thepath; a target traveling the path, wherein the path is infrequentlytraveled; a target traveling at an unusual speed on the path; a targetstopping at an unusual stopping point on the path; a target traveling atan unusual time on the path; a target traveling in an unusual directionon the path; a target of a type not normally found on the path; or atarget having an unusual physical property on the path.
 27. Thecomputer-readable medium according to claim 23, the operations furthercomprising predicting a target's subsequent path based on said at leastone path model and the target's observed behavior.
 28. Thecomputer-readable medium according to claim 23, wherein said analyzingand identifying target behavior includes classifying at least one of thetarget type, based on a path type, or the path type based on the targettype.
 29. The computer-readable medium according to claim 23, whereinsaid analyzing and identifying target behavior further includes buildinga statistical model of target behavior with respect to the path model.30. The computer-readable medium according to claim 29, wherein saidanalyzing and identifying target behavior includes analyzing targetbehavior with respect to a statistical model of at least one of: atarget dwell time duration on the path; a target dwell location on thepath; a target property associated with at least one of: the path, adwell location on the path, and/or a dwell time on the path; aninteraction of a target with another target of the same type on thepath; an interaction of a target with another target of a different typeon the path; an interaction of a target with an element in the scene; atemporal pattern of a target property on the path; or a deviation fromnormal target properties on the path.
 31. The computer-readable mediumaccording to claim 30, wherein said analyzing and identifying targetbehavior includes identify target behavior from a combination of: atleast two detected behaviors inconsistent with a path model; at leasttwo target detected behaviors with respect to a statistical model; or atleast one detected inconsistent behavior and at least one detectedbehavior with respect to a statistical model.